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1.  Introduction 


1.1  Problem  and  Opportunity 

Rapid  advances  in  many  technology  areas  will  soon  make  possible  small  or  micro  air 
vehicles  that  can  be  deployed  in  large  numbers  at  low  cost.  In  theory,  military  and  commercial 
applications  of  such  vehicles  abound.  However,  in  practice  these  vehicles  must  achieve  a  high 
level  of  autonomy  in  operations  to  be  successfully  deployed  in  large  numbers.  Furthermore, 
much  of  their  utility  is  to  be  derived  from  operations  in  densely  cluttered  environments  that 
include  unknown  obstacles.  Traditional  guidance  and  control  technology  is  inadequate  to  meet 
the  demand  for  autonomy  in  such  environments,  and  dependence  on  traditional  design  methods 
will  lead  to  expensive  solutions  that  demand  tremendous  computational  resources.  There  are, 
however,  a  multitude  of  biological  systems  that,  with  relatively  crude  sensors  and  limited 
computing  resources,  operate  very  successfully  in  such  environments.  This  program  seeks  to 
emulate  the  success  of  these  biological  systems  in  an  engineered  guidance  and  control  system. 
This  is  to  be  accomplished  by  developing  a  complete  understanding  of  the  tracking,  guidance  and 
control  functions  of  a  candidate  biological  system  (the  flesh  fly),  and  to  then  use  this 
understanding  to  inspire,  develop  and  demonstrate  effective  and  affordable  guidance  and  control 
technologies  for  autonomous  operation  of  small  or  micro  air  vehicles. 

The  STTR  program  solicitation  issued  by  the  Air  Force  calls  for  a  demonstration  of  “the 
use  or  incorporation  of  insect  neurobiology  or  sensory  processing  into  the  design  of  precision 
flight  capability  of  autonomous  air  vehicles.”  More  specifically,  the  solicitation  calls  for 
development  and  demonstration  of  biologically  inspired  guidance,  navigation  and  control  (GNC) 
sensors,  components,  and/or  systems  that  will  enable  an  autonomous  small  air  vehicle  to  search 
for,  detect,  pursue,  and  rendezvous  with  an  evasive  target  in  a  densely  cluttered  environment. 

As  stated  in  the  solicitation,  desirable  characteristics  of  the  GNC  system  include: 

•  Tolerance  to  transient  sensor  information  distortion  or  obscuration; 

•  Capable  of  obstacle  avoidance  (e.g.  buildings  or  trees,  power  lines,  or 
overhanging  branches); 

•  Detection,  acquisition,  tracking  and  guidance  to  a  moving  target  in 
background  clutter; 

•  Precision  rendezvous  with  the  target  (e.g,  warhead  event,  electronic  tagging); 

•  Flight  control  robustness  to  wind  gusts  and  turbulence  near  structures. 


Guided  Systems  Technologies,  Inc.  (GST)  assembled  a  uniquely  qualified  team  of 
researchers  to  address  these  requirements.  The  phase  I  program  was  designed  to  demonstrate  the 
feasibility  and  merit  of  combining  university  research  in  the  fields  of  entomology,  target  tracking, 
aerospace  vehicle  guidance  and  control,  as  well  as  image  processing  and  computer  vision,  with 
the  practical  systems  integration  and  UAY  capabilities  to  meet  the  challenge.  The  phase  I 


1 


program  produced  significant  results  in  three  distinct  areas  that  are  to  be  extended  and  combined 
in  phase  II  to  produce  a  demonstration  of  the  solicited  capability, 

1.2  Organization  of  the  Report 

Chapter  2  presents  a  summary  of  neural  network  adaptive  output  feedback  control  and 
illustrates  it  use  in  design  of  a  high  bandwidth  flight  control  system  for  unmanned  helicopters. 

Chapter  3  presents  a  novel  method  to  extend  the  existing  neural  network  adaptive  control 
methods  into  the  realm  of  intercept  guidance  law  development.  The  chapter  begins  with  a  review 
of  the  classic  proportional  navigation  guidance  law.  An  adaptive  form  of  the  guidance  law  is 
then  developed.  The  chapter  concludes  with  the  presentation  of  simulation  results  for  intercept 
of  a  target. 

Chapter  4  documents  the  phase  I  work  completed  with  regard  to  experimental 
characterization  of  flesh  fly  pursuit  and  evasion.  The  chapter  includes  a  description  of  the 
experimental  methods  and  facilities,  as  well  as  presentation  of  results  obtained  to  date. 

Chapter  5  presents  a  compilation  of  work  performed  in  the  area  of  image  processing. 

Chapter  6  completes  the  report  with  a  summary  of  the  phase  I  effort,  draws  conclusions 
from  the  work  completed,  and  makes  recommendations  for  continued  research. 
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2.  Background  on 
High  Bandwidth  Adaptive  Control 


2.1  Background  on  Neural  Network  Augmented  Control. 

Researchers  at  the  Georgia  Institute  of  Technology  and  GST  have,  over  the  last  decade, 
developed  very  effective  methods  for  realizing  practical  neural  network  adaptive  control  systems 
for  flight  vehicles.  For  example,  in  the  USAF  RESTORE  (reconfigurable  systems  for  tailless 
fighter  aircraft)  program,  an  adaptive  command  augmentation  system  based  on  full  state  feedback 
was  used  to  stabilize  an  airframe  and  maintain  acceptable  handling  qualities  in  a  wide  variety  of 
simulated  control  surface  failures,  and  was  successfully  flight  tested  on  the  X-36  unmanned  test 
aircraft.  This  same  adaptive  control  technology  has  been  applied  to  autopilot  design  for  precision 
guided  munitions,  and  was  flight  demonstrated  on  the  MK-84  JDAM  in  2001.  GST  and  Georgia 
Tech  also  have  an  active  program  to  flight  test  a  recently  developed  output  feedback  formulation 
of  the  neural  network  adaptive  law  on  a  small  agile  unmanned  helicopter.  The  output  feedback 
result,  when  combined  with  a  technique  known  as  pseudo  control  hedging,  allows  for  the  design 
of  a  very-high  bandwidth  controller  that  can  be  used  to  exploit  the  full  operational  envelop  of  the 
unmanned  flight  vehicle  in  a  manner  similar  to  that  achieved  by  a  skilled  human  pilot.  This 
neural  network  adaptive  control  system  technology  is  a  key  element  of  the  foundation  on  which 
the  subject  work  of  adaptive  guidance  is  based,  and  is  therefore  reviewed  in  some  detail  in  this 
chapter. 

2.2  Review  of  Neural  Network  Adaptive  Control 

Both  classical  and  modem  control  design  methods  are  fundamentally  limited  by  the 
presence  of  unmodeled  high  frequency  effects,  and  control  nonlinearites  due  to  saturation.  The 
same  is  true  in  adaptive  methods  that  do  not  attempt  to  learn  and  interact  with  these  effects.  Yet 
biological  systems  routinely  exhibit  high  bandwidth  performance  in  the  presence  of  these  effects. 
Therefore,  one  goal  of  the  proposed  research  in  control  is  to  demonstrate  an  approach  to  flight 
control  design  that  is  inspired  by  the  performance  observed  in  nature.  The  approach  is  illustrated 
below  using  results  for  the  design  of  pitch-angle  flight  control  system  of  an  experimental 
unmanned  helicopter. 

This  section  presents  a  novel  approach  to  adaptive  output  feedback  control1 .  Of  particular 
interest  here  is  its  use  in  high  bandwidth  flight  control,  which  is  made  possible  through 
interaction  with  (as  opposed  to  avoidance  of)  poorly  modeled  high  frequency  dynamics,  and 
saturation.  Adaptation  is  achieved  using  only  input/output  sequences  of  the  uncertain  system. 
Furthermore,  adaptation  continues  under  saturation  (position  and/or  rate)  and  the  methodology 
permits  other  strong  nonlinearities  that  may  due  to  mixed  utilization  of  continuous  and  discrete 
(bang-bang)  control. 
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Neural  Network-Based  Adaptive  Output  Feedback  Control:  Consider  the  dynamics  of  an 
observable  nonlinear  plant  represented  by  the  following  non-affine  SISO  system 

x  =  f{x,u),y  =  h{x)  (1) 

where  xeR",  u, y  e  R  and  f,he  C00  may  be  partially  known  or  unknown  functions.  Assume 
that  n  and  the  relative  degree(r)  are  known.  With  the  knowledge  of  n  and  r,  we  can  postulate  an 
equivalent  expression  for  the  system  in  a  normal  form  as 

y{r)  =  hr(0,T],u) 

where  0  =  (y, y{r~l))T  eRr,  and  7  e  R"r  is  the  state  vector  associated  with  the  zero 
dynamics 

=  (3) 

which  are  assumed  to  be  stable.  It  is  often  the  case  that  an  approximate  expression  exists  for 
hr  (■')  in  Eq.(2)  which  can  be  formulated  using  only  the  instantaneous  available  measurements. 
This  approximation  can  be  used  as  a  basis  for  an  inverting  control  law  by  defining 

K(y,u)  =  v  (4) 

where  v  is  the  so-called  pseudo  control  variable.  It  is  assumed  that  this  function  can  be  inverted 
to  obtain  the  actual  control  in  the  form 


u  =  K\y,v)  (5) 

Applying  the  approximate  inverse  into  Eq.(2),  we  obtain 

y(r)  -  v  +  A  (6) 

where  A  =  hr{<p,u)-hr{y,u) . 


Figure  2.1  depicts  the  structure  of  this  closed-loop  neural  network-based  adaptive  control 
system.  The  adaptive  control  term,  vad ,  is  introduced  to  cancel  the  model  inversion  error  A . 

From  the  universal  approximation  property  of  multilayer  NNs,  a  single-hidden  layer  NN  is 
selected  to  approximate  A .  The  structure  of  the  NN,  presented  in  Figure  2.2,  is  expressed  as 
WTa(VTx )  where  W,  V  are  matrices  containing  the  interconnection  weights,  and  <x(*) ,  shown 
in  Figure  2.3,  is  an  activation  function  of  the  hidden  nodes. 

The  following  weight  update  rules  in  Equation  (7)  guarantee  the  uniform  ultimate 
boundedness  of  the  error  dynamics  and  NN  weights. 
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(7) 


V  =  -Yv[r){zWTa')+Avv) 

&  =  -r J[&  -  af Trt)z  +  Xww] 


Control  Hedging:  In  order  to  exploit  the  full  maneuvering  capability  of  a  flight  vehicle 
with  adaptive  control  system  design,  it  is  necessary  to  protect  the  adaptive  process  from  adapting 
to  characteristics  such  as  control  saturation.  The  approach  employed  here  is  referred  to  as 
Pseudo-control  Hedging.  In  this  approach,  the  adaptation  is  protected  from  the  effects  of  hard- 
nonlinearities  by  shifting  the  dynamics  in  the  command  filter  portion  of  the  control  system 
architecture  backwards  by  an  estimate  of  the  amount  that  the  real  system  is  unable  to  achieve  due 
to  hard  nonlinearities. 

This  approach  can  be  used  to: 

•  Isolate  the  effects  of  hard  nonlinearities  from  the  adaptation 

•  Mitigate  the  effects  of  approximately  known  input  dynamics  (such  as  actuator  dynamics) 
from  the  adaptation 

•  Allow  the  adaptation  to  handle  a  class  of  unknown  hard  nonlinearities  (such  as  an  actuator 
failure). 


Figure  2.1  Block  diagram  of  neural  network  adaptive  controller  -  the  output 
feedback  formulation  without  an  observer. 
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Hidden  Layer 

Figure  2.2  Topology  of  a  single-hidden  layer  neural  network  and  a  typical  activation  function. 


Figure  2,3  Typical  activation  function  employed  in  each  node  of  the  hidden  layer,  plotted  for 
different  values  of  the  internal  activation  potential,  a. 


Design  Application  to  Pitch-attitude  Control  of  an  Unmanned  Air  Vehicle:  Consider  the 
design  of  a  flight  control  system  for  a  small  unmanned  helicopter.  In  the  practical  design  of  such 
a  vehicle  it  is  common  to  include  a  small  control  rotor.  In  this  case,  control  inputs  to  the  main 
rotor  are  instead  fed  to  the  control  rotor,  and  the  main  rotor  control  inputs  are  a  result  of  the 
control  rotors  reaction  to  both  command  and  flight  conditions.  The  control  rotor  is  thus  a 
mechanical  feedback  device  and  can  be  used  by  the  designer  to  impart  desired  characteristics  on 
the  response  of  the  overall  dynamic  system.  While  these  same  characteristics  can  be  imparted 
through  feedback  with  an  electronic  control  system,  it  is  desirable  to  retain  the  control  rotor  for 
manual  flight  operations  such  as  may  be  required  in  response  to  an  electronic  system  failure  or 
software  fault  on  an  experimental  control  system  testbed.  The  control  input  to  the  main  rotor  is 
affected  by  the  control  rotor  flapping  angle,  actuator  dynamics  and  time  delay  which  add 
additional  dynamics  within  the  desired  bandwidth  of  the  control  system  design.  The  pitch 
channel  equation  of  motion  of  such  a  helicopter  can  be  expressed  as  a  following  SISO  linearized 
system: 


u 
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where  y  is  the  controlled  variable.  The  state  vector  x  =  (w,  q,  9,  pY  where  u  is  forward  velocity,  q 
is  pitch  rate,  6  is  pitch  angle,  P  is  control  rotor  flapping  angle  and  the  control  variable  8  is  the 
longitudinal  cyclic  input.  The  bandwidth  limitation  is  largely  due  to  control  rotor  dynamics  (at 
approximately  8-10  rad/s),  and  filtering  and  digital  processing  time  delays  (between  0.02  to  0.04 
seconds)  that  are  present  in  the  control  path  from  the  rate  sensor  to  the  main  rotor  swash  plate. 
Moreover,  there  are  significat  position  and  rate  limits  in  the  actuator.  The  approximate  inversion 
law 


=  h  1  (q,  v)  =  -4—  («v  -  Mq  (Mq  +  a)q)  (9) 

M  s 

y(s}  cc 

corresponds  to  having  the  desired  open  loop  transfer  function  Gd  (5)  =  X±-L  =  - - . 

v(s)  s  (5  +  a) 

The  control  deflection  is  assumed  to  respond  to  the  input  command  with  0.03  second  time  delay 
and  actuator  dynamics  where  its  time  constant  is  selected  as  0.04  sec.  The  control  design  locates 
the  poles  of  the  error  dynamics  at  -8+  6i,  -20.  This  is  well  within  the  bandwidth  of  the  flapping 
dynamics,  control  rotor  dynamics,  and  at  a  frequency  that  results  in  significant  phase  shift  due  to 
time  delay. 

Figure  2.4  compares  the  pitch  attitude  tracking  performance  with  and  without  the  neural  network. 
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The  tracking  performance  without  adaptive  neural  network  results  in  a  limit  cycle  of  the  tracking 
error.  On  the  other  hand,  excellent  tracking  performance  is  achieved  using  the  adaptive  neural 
network.  Moreover,  the  rate  limit  is  encountered  whenever  there  is  a  reversal  in  the  command. 
This  result  exhibits  the  fact  that  the  adaptive  controller  achieves  improved  performance  by 
interacting  with  the  control  rotor  dynamics,  actuator  dynamics,  time  delay  and  control  limits 
This  interaction  is  a  result  of  a  highly  nonlinear  process,  and  is  difficult  to  achieve  using  a  linear 
controller. 


Pitch  Angle 


Figure  2.4  Pitch  attitude  command  and  response  time  histories  with  and 
without  neural  network  augmentation. 
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3.  Adaptive  Guidance  Law  Development 


A  primary  phase  I  objective  was  to  develop  a  direct  adaptive  approach  for  correcting  a 
proportional  navigation  (PN)  guidance  law  for  uncertainty  associated  with  the  target  acceleration. 
The  term  “direct”  is  used  in  this  context  to  identify  the  fact  that  the  developed  approach  does  not 
depend  on  the  traditional  use  of  an  observer.  Observer-based  solutions  rely  on  a  model  for  target 
behavior,  whereas  a  direct  adaptive  approach  that  is  neural  network  (NN)  based  can  be  free  of 
target  modeling  assumptions,  relying  instead  on  the  universal  approximation  properties  of 
artificial  neural  networks  to  model  the  target’s  evasive  strategy.  Such  an  approach  is  expected  to 
be  less  vulnerable  to  intelligent  targets  that  can  take  advantage  of  the  modeling  assumptions 
imbedded  in  observer-based  design  approaches,  and,  because  it  is  adaptive,  to  better  emulate  the 
anticipated  intercept  behavior  of  biological  systems.  Adaptation  can  also  be  used  to  introduce 
other  benefits  in  the  engineered  guidance  and  control  solution,  such  as  the  ability  to  address 
system  nonlinearities. 


3.1  Review  of  Proportional  Navigation 

A  simple  two-dimensional  target  engagement  scenario  is  depicted  in  Figure  3.1. 


Figure  3.1.  A  planar  engagement  model. 


The  line-of-sight  (LOS)  rate  and  range  rate  for  this  scenario  are  given  by: 

a  _  x  _  Yi  sin(^  z3zXk  sin(^.  -  j) 

R  (3.1) 

R  =  Vt  cos (6t  —  A)  —  Vm  cos(dm  -  X) 

Defining  am  as  the  missile  acceleration  normal  to  the  LOS,  then 


To  simplify  the  analysis  we  introduce  the  following  two  assumptions: 

Assumption  1 :  The  velocities  are  constant 
Assumption  2:  The  range  rate  is  constant 

Differentiating  the  first  of  (3.1),  and  using  R=-xR  ,  where  x  denotes  time-to-go,  and  invoking  the 
above  assumptions  we  obtain 


db  =  2co/T  +  (aT-am)/R  (3.3) 

3.2  Required  Acceleration 

Substituting  the  PN  guidance  law, 

am  =  -N  R  co  =  NVcco  (3.4) 

where  Vc  is  the  closing  velocity,  gives 

cb  =  (2-N)o)/T  +  aT/R  (3.5) 

It  is  well  known  that  PN  guidance  with  navigation  gain,  N  =  3,  is  optimal  in  the  case  where  a r= 
0  (non-maneuvering  target)2.  Optimality  is  with  respect  to  minimizing  the  integral  square  of  the 
acceleration,  with  intercept  formulated  as  a  terminal  constraint.  The  resulting  acceleration 
against  a  constant  accelerating  target  is  given  by 

(3-6) 

Figure  3.2  illustrates  the  normalized  acceleration  resulting  from  (3.6)  as  it  depends  on  N  and 


Augmented  PN  guidance  (APN)  is  optimal  for  constant  target  maneuvers,  and  is  given  by 

am  =  N(Rco  +  ar/2)  (3.7) 

The  resulting  acceleration  against  a  constant  accelerating  target  is  given  by 

am/aT=0.5N(\-t/tf)N~2  (3.8) 

Figure  3.3  illustrates  the  normalized  acceleration  for  this  guidance  law.  Note  that  the 
acceleration  profiles  are  monotonically  increasing  functions  of  t/tf  for  PN,  and  the  reverse  is  true 
for  APN.  Bear  in  mind  that  for  N  =  3  and  a t  —  constant,  the  APN  law  is  optimal,  whereas  the 
PN  law  is  not. 
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The  above  laws,  when  implemented  in  their  ideal  forms,  result  in  zero  miss  against  most 
forms  of  evasive  target  maneuvers.  Real  implementations  result  in  non-zero  miss  distances  due 
to  a  combination  of  factors.  The  primary  sources  of  miss  are:  lags  within  the  guidance  loop, 
nonlinearities,  sensor  noise  and  other  sources  of  tracking  error  such  as  radome  slope  error. 
Sensor  errors  also  place  a  lower  limit  on  the  effective  time  constant  of  the  guidance  loop3.  An 
illustration  of  the  miss  that  results  when  a  first  order  lag  with  a  time  constant  of  T  =  1.0  sec  is 
modeled  in  the  guidance  loop,  is  given  in  Figure  3.4.  This  shows  the  miss  due  to  a  constant  3G 
target  acceleration  as  a  function  of  time-to-go  at  the  start  of  the  maneuver. 

While  Figure  3.4  illustrates  the  effect  of  a  constant  target  maneuver,  it  can  be  shown  that 
the  optimal  evasive  maneuver  is  bang-bang,  with  reversals  occurring  at  times-to-go 
corresponding  to  the  peaks  in  the  above  curves3.  So  for  N  =  3,  the  optimal  evasive  strategy  is  to 
reverse  the  maneuver  at  approximately  x=  2.5  seconds.  This  would  have  the  net  effect  of 
doubling  the  peak  miss  distance,  or  achieving  a  total  miss  of  approximately  54  feet,  Kalman 
filters  have  traditionally  been  used  in  the  guidance  loop  to  estimate  target  acceleration,  in  order  to 
implement  APN.  While  this  is  very  effective  against  target  maneuvers  that  belong  to  the  class 
for  which  the  filter  is  designed,  it  also  makes  the  overall  design  more  vulnerable  to  evasive 
strategies  that  take  advantage  of  these  modeling  assumptions. 


Figure  3.4.  Miss  distance  (feet)  versus  time-to-go  at  launch  due  to  a  3G  target  maneuver 
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3.4  Neural  Network-Based  Adaptive  Guidance 


There  are  four  (4)  major  challenges  in  considering  the  application  of  adaptive  control  to 
guidance  problems.  The  first  concerns  the  fact  that  guidance  problems  are  fundamentally  finite 
time  problems,  whereas  adaptive  control  is  formulated  as  a  problem  in  stability  for  an  infinite 
time  horizon.  That  is,  Lyapunov  stability  analysis  is  employed  to  derive  or  justify  the  adaptive 
laws,  and  the  focus  is  on  stability  of  an  equilibrium  point,  normally  taken  as  the  origin  in  the 
error  coordinates.  One  approach  (developed  in  phase  I)  to  overcome  this  problem  is  to  consider 
the  following  time  transformation 


ti  =  In  (1/t)  (3.9) 

where  x  =  tf  -t.  This  transforms  the  LOS  rate  dynamics  in  (3.3)  to  the  following  linear,  time 
invariant  form 


dcol  dtx  =2 <B  +  {am  -aT)l R 


(3.10) 


This  suggests  that  the  adaptive  control  problem  should  be  formulated  in  the  ti  time  scale. 
The  adaptive  laws  would  be  integrated  in  the  real  time  scale,  but  with  an  integration  step  of  dti  = 
dt/x,  where  dt  is  the  increment  in  real  time. 


The  second  difficulty  surrounds  the  fact  that  the  guidance  problem  is  fundamentally  an 
output  feedback  problem.  The  adaptive  output  feedback  problem,  and  its  application  in  design  of 
a  high-bandwidth  autopilot  were  reviewed  in  Chapter  2.  In  particular,  this  direct  approach  to 
adaptive  output  feedback  design  avoids  the  introduction  of  a  model-based  observer4.  This 
formulation  is  also  applicable  to  problems  with  unmodeled  dynamics,  however  this  issue  is 
beyond  the  scope  of  the  discussion  in  this  chapter.  However,  a  drawback  to  the  method  of 
Chapter  2  and  Ref.  3  is  that  it  is  based  on  feedback  inversion  of  the  plant  dynamics,  which 
constitutes  the  third  major  barrier  to  be  surmounted.  Feedback  inversion  is  not  appropriate  for 
intercept  guidance  applications  for  numerous  reasons.  Ideally,  we  seek  an  adaptive  approach  that 
augments  a  nominal  intercept  guidance  solution  (such  as  PN)  to  correct  for  evasive  target 
maneuvering.  One  such  architecture,  based  on  fundamental  concepts  developed  under  an 
independent  effort  at  Georgia  Tech,  which  appears  to  be  suitable  for  surmounting  this  difficulty 
is  shown  in  Figure  3.5.  This  architecture  assumes  that  PN  guidance  will  be  augmented  by  an 
adaptive  process  in  such  a  way  that  the  plant  behaves  like  the  model  response,  where  the  model 
response  is  based  on  Equation  (3.10)  with  a t~  0.  In  the  ideal  case,  the  main  differences  between 
these  responses  are  due  to  the  target  acceleration,  and  nonlinearities  in  the  engagement  dynamics, 
part  of  which  may  be  due  to  saturation.  Therefore,  one  role  of  the  adaptive  process  is  to  augment 
the  PN  law  by  canceling  the  effect  of  the  target  maneuver.  We  have  shown  that  the  error 
dynamics  for  this  architecture  fits  the  form  addressed  in  Ref  3,  and  therefore  the  NN  based 
adaptive  control  laws  developed  there  should  be  applicable.  Note  that  the  proposed  adaptive 
guidance  architecture  is  appropriate  for  augmenting  any  nominal  guidance  solution,  such  as 
that  derived from  the  study  of  biological  systems. 
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Hedge  Signal 


Figure  3.5.  Adaptive  guidance  architecture 

The  fourth  technical  challenge  is  to  modify  the  adaptive  process  to  permit  adaptation  to 
continue  during  periods  of  control  saturation.  Adaptive  processes  are  highly  sensitive  to 
saturation,  similar  to  integrator  windup  in  non-adaptive  systems.  Ref.  5  describes  a  method 
called  ‘hedging’,  which  can  be  used  to  prevent  the  adaptive  process  from  attempting  to  cancel 
selected  effects  that  can  be  represented  at  the  input  side  of  the  plant.  These  include  effects  due  to 
saturation,  and  command  limits  that  are  intentionally  inserted  to  avoid  exceeding  airframe  limits. 
Hedging  works  by  moving  the  model  response  backwards  by  an  amount  equal  to  the  deficit  in 
achievable  acceleration.  The  location  where  the  hedging  is  inserted  is  illustrated  in  Figure  3.5. 
In  this  case,  the  hedging  signal  amounts  to  the  difference  between  the  command  missile 
acceleration  (amc  in  Fig.  3.5)  and  the  actual  missile  acceleration.  This  means  that  all  effects  due 
to  vehicle  +  autopilot  dynamics,  including  saturation,  are  hedged.  Thus,  the  NN  is  prevented 
from  adapting  to  these  effects.  Subsequently,  we  may  wish  to  adapt  to  the  airframe  dynamics, 
and  hedge  only  the  effects  of  saturation.  This  will  be  explored  during  the  Phase  II  part  of  this 
effort. 


3.5  Numerical  Results 

We  have  conducted  a  preliminary  test  using  the  architecture  in  Figure  3.5  and  the 
adaptive  control  approach  of  Ref.  4.  A  second  order  model  was  used  to  represent  the  autopilot 
lag  in  the  guidance  loop,  with  ©n  =  2/T  and  C,  =  2/T/  con.  This  choice  results  in  T  being  the 
effective  guidance  loop  time  constant,  in  the  sense  that  the  second  order  model  approximates  the 
rise  time  of  a  first  order  system  with  this  time  constant.  We  chose  T  =  0.5  for  the  study.  For  this 
model,  the  optimal  target  maneuver  against  a  PN  guidance  law  with  N  =  3  is  a  reversal  at  time- 
to-go  =  1.15  seconds,  close  to  the  value  predicted  by  Figure  3.4  for  a  first  order  lag  if  the  time 
axis  is  viewed  as  normalized  by  1/T.  The  PN  guidance  law  alone  resulted  in  a  miss  distance  of 
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41.6  feet.  With  adaptive  guidance,  the  miss  distance  was  reduced  to  2.8  feet.  This  result  may  be 
further  improved  if  we  correctly  account  for  the  relative  degree  of  the  LOS  rate,  thereby 
including  the  effect  of  the  autopilot  lag  in  the  design,  rather  than  simply  hedging  the  acceleration 
command  for  these  dynamics. 

Figure  3.6  compares  the  resulting  acceleration  profiles  in  g’s.  Note  that  the  adaptive 
controller  appears  to  be  effective  in  estimating  target  acceleration.  Following  an  initial  transient, 
its  acceleration  profile  decreases  with  time,  similar  in  form  to  the  ideal  guidance  law  result  that 
has  perfect  knowledge  of  target  acceleration  (see  Figure  3.3).  Note  that  the  initial  response  of  the 
non-adaptive  PN  law  is  similar  to  that  depicted  in  Figure  3.2. 

In  Fig.  3.7  we  examine  the  effect  of  an  acceleration  command  limit  of  5  g’s,  for  the  case 
of  PN  guidance  (N  =  3)  with  an  autopilot  lag  of  0.5  seconds,  and  a  constant  3  g  target  maneuver. 
Because  the  PN  law  results  in  an  acceleration  profile  that  increases  linearly  with  time,  the  limit 
ultimately  results  in  a  miss  distance  of  approximately  209  feet.  The  adaptive  law  results  in 
negligible  miss  distance.  These  results  demonstrate  the  ability  to  correct  for  target  acceleration 
without  the  use  of  an  observer,  or  pre-defined  target  acceleration  model.  The  approach  also 
corrects  the  guidance  law  for  the  effect  of  nonlinear  engagement  dynamics  that  result  in  large 
rotations  of  the  LOS  angle.  However,  this  is  not  apparent  from  the  results  in  this  example. 


Figure  3.6.  Comparison  of  acceleration  profiles. 
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Fig,  3.8  shows  a  comparison  of  the  trajectories  when  subjected  to  a  5  g  limit.  Note  that 
the  trajectory  with  adaptive  guidance  does  a  better  job  of  leading  the  target,  consequently  the 
acceleration  needed  near  the  end  of  the  trajectory  is  significantly  reduced,  making  it  far  less 
vulnerable  to  a  reversal  in  the  target  maneuver.  The  miss  distances  are  not  apparent  at  this  scale. 

Figure  3.9  shows  the  effect  of  reducing  the  missile  acceleration  limit  to  3.5  and  3.0  g’s  in 
the  adaptive  guidance  solution.  Note  that  with  hedging,  adaptive  guidance  behaves  quite  well, 
even  when  the  maneuver  advantage  of  the  missile  is  slightly  greater  than  that  of  the  target. 
However,  when  the  maneuver  capability  is  matched  to  the  target,  then  the  adaptive  solution  is  not 
able  to  compensate  the  PN  law  for  target  acceleration  at  this  initial  engagement  geometry  (head- 
on).  Since,  the  target  can  not  be  intercepted  from  this  condition  even  with  maximum 
acceleration,  it  is  doubtful  that  this  result  can  be  significantly  improved  without  either  accounting 
directly  for  the  lag  in  the  guidance  loop,  or  investigating  the  possibility  of  recovery  after  a  miss 
has  occurred.  The  degree  to  which  this  can  be  done  will  be  one  topic  for  future  investigation. 


Figure  3.9.  Comparison  of  trajectories’  acceleration  profiles  when  the  adaptive 
law  is  limited  to  3.5  and  3.0  g’s. 

In  summary,  a  novel  method  had  been  developed  which  treats  the  problem  of  target 
acceleration  by  incorporating  a  neural  network  in  place  of  an  observer  as  an  element  in  a 
guidance  law.  This  results  in  a  direct  adaptive  approach  for  compensating  a  PN  guidance  law, 
which  should  be  less  sensitive  to  smart  target  evasive  maneuvers.  Preliminary  simulation  results 
indicate  that  the  approach  has  merit. 
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4.  Characterization  of  Flesh  Fly  Pursuits 


The  overall  goal  in  this  area  is  to  fully  understand  how  male  flesh  flies  track  their  targets, 
especially  in  environments  cluttered  with  obstacles  and  alternative  targets  (other  flies),  and 
subsequently  to  use  this  understanding  to  emulate  this  capability  in  an  engineered  system. 
During  Phase  I,  Prof.  Cole  Gilbert  and  Dr,  P.O.  Zanen  of  Cornell  University  developed  a  reliable 
system  for  eliciting  aerial  pursuits  of  females  by  male  flesh  flies  (see  Figure  4.1).  The  current 
flight  arena  (see  Figure  4.2)  has  a  calibrated  active  volume  of  17x17x40cm,  which  is  a 
subvolume  of  a  larger  cage  (50x50x80cm)  in  which  single  males  and  5-8  virgin  females  are 
released  several  days  after  eclosion  as  adults.  Flies  are  15mm  long  or  shorter  with  a  total  mass  of 
70mg  (head:  10.4mg).  As  the  participants  fly  through  the  active  volume,  we  capture  pursuits 
with  a  digitizing  precision  of  1mm.  To  increase  accuracy  in  digitized  orientation  of  the  sensor 
(head)  we  attach  a  small  fiducial  stick  of  balsa  wood  (8mm,  1.6mg)  that  indicates  the  seeker’s 
line-of-sight  (LOS).  Pursuit  sequences  are  captured  with  a  Kodak  MotionCorder  SR500  at  250fps 
and  directly  read  into  the  hard  drive  buffer  of  the  SR500.  Good  sequences  are  archived  onto  Hi-8 
videotape  for  offline  digitizing  and  subsequent  analysis  by  custom  software  [Mantid  32,  Syneeros 
Inc.]  developed  by  Dr.  P.O.  Zanen  in  the  Cornell  group.  This  software  enables  digitization  of 
any  objects  in  the  fields  of  view. 


FLESH  FLY 

Neobellieria  bullata  ('Sarcophagidae') 

body  length:  <  15  mm 

mass:  70  mg 


linear  velocity:  < 
angular  velocity: 


1  m  s' 


1000°s 


0-1 


Figure  4.1,  Male  flesh  fly  under  study. 
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Digital  stereo  imaging  system 
and  light  source  mounted  above 
the  cage. 


Flight  Arena  (cage)  of 
rectangular  volume  with 
screen  covering 


Figure  4.2  Photograph  of  the  current  flight  cage  with  imaging  equipment  mounted  above. 


During  the  phase  I  effort,  the  test  setup  and  associated  procedures  described  above  were 
gradually  improved  to  ultimately  obtain  a  number  of  very  good  pursuit/evasion  trajectories  that 
can  be  analyzed.  Figure  4.3  presents  a  three-dimensional  plot  of  a  “straight”  trajectory  and 
depicts  the  path  of  both  the  male  pursuer  (in  black),  and  the  female  evader  (in  red).  Points  on  the 
plot  represent  position  samples  at  distinct  points  in  time.  The  male  has  been  fitted  with  a  fiducial 
stick  as  described  above  and,  though  hard  to  see  at  this  scale,  the  figure  includes  the  line  of  sight 
angle  by  means  of  a  vector  ascribed  to  the  male  position  at  each  sample  time.  Figure  4.4  presents 
a  second  “curved  trajectory.  Figure  4.5  presents  a  sequence  in  which  the  female  successfully 
evades  the  male,  but  in  which  the  male  successfully  reengages  his  target  to  complete  the  chase. 
Each  of  these  figures  is  accompanied  by  a  digital  image  sequence  which  is  much  more  effective 
for  initial  description  of  the  scenario,  but  which  cannot  be  included  in  a  printed  document  such  as 
this.  The  video  clips  of  these  three  trajectories  were  presented  to  the  sponsor  at  the  phase  I  final 
program  meeting,  and  delivered  on  a  compact  disk.  Also  included  in  Figure  4.5  is  a  plot  of  the 
computed  position  of  the  female  in  the  field  of  view  of  the  male  during  the  pursuit.  Analysis  of 
the  optical  flow  of  this  type  of  data  as  it  evolves  during  the  pursuit  is  planned  for  the  next  phase, 
and  will  reveal  much  about  the  sensory  input  to  the  male  during  the  pursuit. 
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Figure  4.5  Evasive  target  with  correction  by  pursuer  on  left,  and  plot  of 
computed  target  position  in  field  of  view  of  pursueing  male  on  right. 
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The  team  made  significant  progress  in  this  area  in  Phase  I,  and  clearly  demonstrated  the 
feasibility  of  collecting  accurate  three-dimensional  position  and  orientation  data  for  analysis. 
However,  further  improvements  to  the  test  procedure  are  still  needed  to  fully  characterize  the 
engagement.  The  proposed  improvements  are  discussed  in  detail  in  Section  E  of  the  Phase  II 
proposal,  as  are  the  plans  for  experiments  which  will  reveal  the  response  of  the  male  in  pursuit  to 
background  clutter  and  obstacles. 
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5.  Image  Processing  Algorithms 


This  section  summarizes  visual  tracking  techniques  that  have  been  improved  and  applied 
as  part  of  the  Phase  I  effort.  This  work  draws  upon  team  member  Prof.  Allen  Tannenbaum’s 
experience  in  image  processing  and  computer  vision  employing  certain  geometric  variational 
evolution  equations. 

Vision  is  a  key  sensor  modality  in  both  the  natural  and  man-made  domains.  The 
prevalence  of  biological  vision  in  even  very  simple  organisms,  indicates  its  utility  in  man-made 
machines.  More  practically,  cameras  are  in  general  rather  simple,  reliable  passive  sensing 
devices  which  are  quite  inexpensive  per  bit  of  data.  Furthermore,  vision  can  capture 
multispectural  information  at  a  high  rate  at  high  resolution,  and  with  a  wide  field  of  view. 
Finally,  cameras  can  be  used  in  an  active  manner.  Namely,  one  can  include  motorized  lenses 
mounted  on  mobile  platforms  that  can  actively  explore  the  surroundings  and  suitably  adapt  their 
sensing  capabilities.  For  some  time  now,  the  role  of  control  theory  in  vision  has  been  recognized. 
In  particular,  the  branches  of  control  that  deal  with  system  uncertainty,  namely  adaptive  and 
robust,  have  been  proposed  as  essential  tools  in  coming  to  grips  with  the  problems  of  both 
biological  and  machine  vision.  These  problems  all  become  manifest  when  one  attempts  to  use  a 
visual  sensor  in  an  uncertain  environment,  and  to  feed  back  in  some  manner  the  information 
gathered. 

Specifically,  Dr.  Tannenbaum’s  research  has  impacted  the  following  areas: 

(a)  Geometric  and  Statistical  Methods  for  Deformable  Contours: 

One  of  the  key  techniques  in  active  vision  and  tracking  is  that  of  deformable  contours  or 
snakes.  These  are  autonomous  processes  that  employ  image  coherence  in  order  to  track  features 
of  interest  over  time.  Our  general  approach  is  based  on  the  theory  of  geodesics  and  minimal 
surfaces  in  a  conformal  geometry.  We  have  introduced  some  more  sophisticated  features  into 
our  geometric-based  cost  function.  In  particular,  we  incorporate  non-local  information  into  our 
models  based  on  the  use  of  adaptive  filtering  schemes  and  Bayesian  statistics.  This  leads  to  a 
natural  synthesis  of  our  work  in  knowledge-based  segmentation  and  active  contours. 

(b)  Area-Preserving  Mappings  and  Optical  Flow: 

The  computation  of  optical  flow  has  proven  to  be  an  important  tool  for  problems  arising 
in  active  vision,  including  visual  tracking.  The  optical  flow  field  is  defined  as  the  velocity  vector 
field  of  apparent  motion  of  brightness  patterns  in  a  sequence  of  images.  It  is  assumed  that  the 
motion  of  the  brightness  patterns  is  the  result  of  relative  motion,  large  enough  to  register  a 
change  in  the  spatial  distribution  of  intensities  on  the  images.  We  have  considered  various 
constrained  optimization  approaches  for  the  purpose  of  accurately  computing  optical  flow.  In 
phase  I,  we  have  weakened  the  optical  flow  constraint  with  one  based  on  the  theory  of  area¬ 
preserving  mappings.  We  want  to  also  include  an  optical  flow  term  in  our  active  contour 
functional  to  better  track  moving  images. 
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(c)  Optimal  Transport  and  Image  Registration: 

Image  registration  is  the  process  of  establishing  a  common  geometric  reference  frame 
between  two  or  more  data  sets  from  the  same  or  different  imaging  modalities  taken  at  different 
times.  This  is  essential  for  data  fusion,  since  registration  can  provide  automated  methods  that 
align  multiple  data  sets  with  each  other  and  with  a  given  object  of  interest.  We  have  been  using 
these  methods  for  data  fusion  as  well  as  a  new  approach  to  optical  flow  for  tracking.  Our 
measure  of  similarity  can  be  based  on  comparing  the  mass  densities  of  the  images. 

We  now  summarize  some  of  our  findings. 

Segmentation.  One  of  the  key  problems  in  vision  is  segmentation,  i.e,,  the  partitioning 
of  an  image  into  homogeneous  regions  with  "homogeneity"  defined  in  various  contexts,  e.g., 
texture,  intensity,  color,  etc.  Numerous  approaches  have  been  proposed  for  this.  We  have 
considered  methods  using  a  synthesis  of  partial  differential  equation  methods  based  on  the 
calculus  of  variations  and  statistical  based  ideas. 

We  have  used  a  combination  of  anisotropic  diffusion  and  the  Bayesian  methods  for  the 
segmentation  of  such  difficult  noisy  imagery  such  as  SAR  (synthetic  aperture  radar)  and 
ultrasound.  In  our  approach,  we  introduce  a  priori  knowledge  about  the  number  of  objects 
present  in  the  image  via  Bayes*  rule.  Posterior  probabilities  obtained  in  this  way  are  then 
amsotropically  smoothed,  and  the  image  segmentation  is  obtained  via  MAP  classifications  of  the 
smoothed  data. 

The  model  we  employ  begins  with  the  assumption  that  the  image  is  composed  of  n 
classes  of  objects.  The  goal  of  our  segmentation  is  to  determine  to  which  class  each  pixel  in  the 
image  belongs.  We  assume  that  the  value  of  each  pixel  in  a  given  class  can  be  thought  of  as  a 
random  variable  with  a  known  normal  distribution,  and  that  these  variables  are  independent 
across  pixels.  Given  a  set  of  intensity  distributions  and  priors,  we  can  apply  Bayes'  Rule  from 
elementary  probability  theory  to  calculate  the  posterior  probability  that  a  given  pixel  belongs  to  a 
particular  class,  given  its  intensity.  We  can  then  calculate  the  posteriors  P,  and  then  to  apply 
anisotropic  smoothing  to  each  P.  Specifically,  we  have  chosen  to  smooth  by  evolving  P 
according  to  a  discretized  version  of  the  affine  geometric  heat  equation.  This  particular  diffusion 
equation  was  chosen  because  of  its  affine  invariance,  because  it  preserves  edges  well,  and 
because  of  its  numerical  stability.  The  final  segmentation  is  obtained  using  the  maximum  a 
posteriori  (MAP)  probability  estimate  after  anisotropic  smoothing. 

These  methods  fit  very  naturally  into  our  overall  research  paradigm  of  combining  curve 
evolution  with  statistical  techniques.  They  will  be  carefully  implemented  and  tested  in  phase  II 
in  order  to  develop  the  optimal  combination  of  the  statistical  and  variational-based  segmentation 
approaches.  Furthermore,  temporal  information  may  be  easily  included  in  this  scheme  as  is 
shown  by  the  images  below  (Figure  5.1).  These  images  were  generated  in  the  phase  I  effort  by 
processing  computer  generated  imagery  from  a  real-time  missile/target  simulation  as  is  proposed 
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for  the  hardware-in-the-loop  testing  in  Phase  II.  Clearly  identified  and  tracked  in  the  image  are 
the  horizon  and  the  target  (both  indicated  in  red). 


Figure  5.1  Two  frames  of  automatic  tracking  of  a  fixed-wing  UAV  target  in 
flight  using  active  contours  with  statistical  methods. 


Optical  Flow.  The  computation  of  optical  flow  is  an  important  tool  for  problems  arising 
in  active  vision.  The  optical  flow  field  is  the  velocity  vector  field  of  apparent  motion  of 
brightness  patterns  in  a  sequence  of  images.  One  assumes  that  the  motion  of  the  brightness 
patterns  is  the  result  of  relative  motion,  large  enough  to  register  a  change  in  the  spatial 
distribution  of  intensities  on  the  images.  Thus,  relative  motion  between  an  object  and  a  camera 
can  give  rise  to  optical  flow.  Similarly,  relative  motion  among  objects  in  a  scene  being  imaged 
by  a  static  camera  can  give  rise  to  optical  flow. 

A  typical  method  is  to  consider  a  spatiotemporal  differentiation  method  for  optical  flow. 
Even  though  in  such  an  approach,  the  optical  flow  typically  estimates  only  the  isobrightness 
contours,  it  has  been  observed  that  if  the  motion  gives  rise  to  sufficiently  large  intensity  gradients 
in  the  images,  then  the  optical  flow  field  can  be  used  as  an  approximation  to  the  real  velocity 
field  and  the  computed  optical  flow  can  be  used  reliably  in  the  solutions  of  a  large  number  of 
problems.  Thus,  Qptical  flow  computations  have  been  used  quite  successfully  in  problems  of 
three-dimensional  object  reconstruction,  and  in  three-dimensional  scene  analysis  for  computing 
information  such  as  depth  and  surface  orientation.  In  object  tracking  and  robot  navigation, 
optical  flow  has  been  used  to  track  targets  of  interest.  Discontinuities  in  optical  flow  have 
proved  an  important  tool  in  approaching  the  problem  of  image  segmentation. 

One  constraint  which  has  often  been  used  in  the  literature  is  the  “optical  flow  constraint" 
(OFC).  The  OFC  is  a  result  of  the  simplifying  assumption  of  constancy  of  the  intensity,  E  = 
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E(x,y,t),  at  any  point  in  the  image.  It  can  be  expressed  as  the  following  linear  equation  in  the 
unknown  variables  u  and  v: 


Et  +  Exu  +  Eyv  =  0 . 

Here  u  and  v  are  the  x  and  y  velocity  components  of  the  apparent  motion  of  brightness  patterns  in 
the  images,  respectively.  It  has  been  shown  that  the  OFC  holds  provided  the  scene  has 
Lambertian  surfaces  and  is  illuminated  by  either  a  uniform  or  an  isotropic  light  source,  the  3-D 
motion  is  translational,  the  optical  system  is  calibrated  and  the  patterns  in  the  scene  are  locally 
rigid. 


It  is  not  difficult  to  see  that  computation  of  optical  flow  is  unique  only  up  to  computation 
of  the  flow  along  the  intensity  gradient  of  E.  This  is  the  celebrated  aperture  problem.  One  way 
of  treating  the  aperture  problem  is  through  the  use  of  regularization  in  computation  of  optical 
flow,  and  consequently  the  choice  of  an  appropriate  constraint.  A  natural  choice  for  such  a 
constraint  is  the  imposition  of  some  measure  of  consistency  on  the  flow  vectors  situated  close  to 
one  another  on  the  image. 

The  optical  flow  constraint  above  is  of  course  very  strong.  Motivated  by  work  on  area¬ 
preserving  maps,  we  have  considered  in  phase  I  a  modification  that  could  be  placed  in  a 
variational  setting.  Indeed,  given  a  family  of  nowhere-zero  2-forms  r' we  have  an  explicit 
method  to  determine  a  family  of  diffeomorphisms  <p‘  such  that 

Differentiating  this  expression  yields  an  expression  very  similar  in  form  to  the  standard  optical 
flow  constraint  with  a  divergence  term  added.  Optical  flow  computed  with  respect  to  this  new 
expression  seems  to  give  rather  encouraging  results  as  is  illustrated  by  Figure  5.2.  These  are  data 
frames  generated  in  Prof.  Gilbert’s  lab  during  Phase  I  and  are  images  of  a  male  flesh  fly  in 
pursuit  of  a  female  within  a  caged  area  and  including  another  female.  The  red  vectors  represent 
the  optical  flow  field  associated  with  the  motion  of  the  three  flies. 

Optimal  Transport,  Image  Registration,  and  Optical  Flow.  In  this  section,  we 
propose  a  method  for  image  registration  and  warping  based  on  the  classical  problem  of  optimal 
mass  transport.  The  mass  transport  problem  was  first  formulated  by  Gaspar  Monge  in  1781,  and 
concerns  the  optimal  way  (in  the  sense  of  minimal  transportation  cost)  of  moving  a  pile  of  soil 
from  one  site  to  another.  This  problem  was  given  a  modem  formulation  in  the  work  of 
Kantorovich,  and  so  is  now  known  as  the  Monge-Kantorovich  problem. 

This  type  of  problem  has  appeared  in  econometrics,  fluid  dynamics,  automatic  control, 
transportation,  statistical  physics,  expert  systems,  and  meteorology.  It  also  naturally  fits  into 
certain  problems  in  computer  vision.  In  particular,  for  the  general  tracking  problem,  a  robust  and 


25 


.  -V,  >|;  . 


reliable  object  and  shape  recognition  system  is  of  major  importance.  A  key  method  to  carry  this 
out  is  via  template  matching,  that  is  one  wants  to  match  some  object  to  a  given  catalogue  of 
objects.  Typically,  the  match  will  not  be  exact  and  hence  some  criterion  is  necessary  to  measure 
the  "goodness  of  fit."  The  matching  criterion  can  also  be  considered  a  shape  metric  for  deciding 
the  similarity  of  one  shape  to  another. 


Figure  5.2  Two  frames  of  automatic  tracking  of  flies  using  optical  flow. 
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We  are  interested  in  applying  these  techniques  to  the  problem  of  registration.  Indeed, 
many  registration  problems  can  be  formulated  in  terms  of  mapping  densities.  The  key  idea  is  to 
find  the  optimal  mapping  via  an  equivalent  problem  involving  certain  factorizations  (called 
"polar")  of  mass-preserving  mappings.  We  have  done  this  via  a  natural  gradient  descent 
technique  similar  to  the  methods  that  we  considered  for  area-preserving  maps  of  minimal 
distortion. 

It  is  important  to  note  that  there  are  a  number  of  ways  in  which  multimodal  registration 
impacts  image-guided  tracking  and  control  systems.  First,  it  allows  for  fusing  the  different  types 
of  information  of  each  imaging  modality,  providing  better  and  more  accurate  information  than 
each  image  viewed  separately.  Further,  it  allows  quantitative  comparison  of  images  taken  at 
different  times,  from  which  precise  information  about  evolution  over  time  can  be  inferred. 
Third,  it  allows  for  updating  in  real  time  a  pre-existing  image  or  model  with  dynamic  data. 

Multimodal  registration  is  in  fact  not  one  but  many  problems.  It  greatly  depends  on  the 
imaging  modalities  to  be  matched,  on  the  objects  being  imaged,  and  the  requirements  of  the 
given  task.  These  parameters  determine  the  assumptions  and  technical  requirements,  which  in 
turn  greatly  influence  the  choice  of  models  and  solution  methods. 

Multimodal  registration  proceeds  in  several  steps.  First,  each  image  or  data  set  to  be 
matched  should  be  individually  calibrated,  corrected  for  imaging  distortions  and  artifacts,  and 
cleared  of  noise.  Next,  a  measure  of  similarity  between  the  data  sets  must  be  established,  so  that 
one  can  quantify  how  close  one  image  is  from  another  after  transformations  are  applied.  Such  a 
measure  may  include  the  similarity  between  pixel  intensity  values,  as  well  as  the  proximity  of 
predefined  image  features  such  as  fiducials,  landmarks,  surface  contours,  and  ridge  lines.  Next, 
the  transformation  that  maximizes  the  similarity  between  the  transformed  images  is  found.  Often 
this  transformation  is  given  as  the  solution  of  an  optimization  problem  where  the  transformations 
to  be  considered  are  constrained  to  be  of  a  predetermined  class.  Once  the  transformation  is 
obtained,  it  can  be  used  to  fuse  the  image  data  sets  in  order  to  plan  a  task,  to  navigate,  or  to  track. 

In  our  case,  we  are  comparing  densities  on  images  via  the  Kantorovieh-Wasserstein 
distance.  This  occurs,  e.g.,  in  functional  imaging  where  one  wants  to  compare  mass  densities  of 
various  features  deforming  over  time,  and  obtain  the  corresponding  elastic  warp  map.  It  also 
allows  us  to  compare  given  scalar  and  vector  fields  on  images. 

There  have  been  a  number  of  algorithms  considered  for  computing  an  optimal  transport 
map.  For  example,  methods  have  been  proposed  based  on  linear  programming  (for  measures 
which  are  weighted  sums  of  delta  functions)  and  on  Lagrangian  mechanics  closely  related  to 
ideas  from  fluid  flows  (Euler  equation,  etc.)  In  our  work,  we  have  used  a  gradient  descent 
approach  based  on  the  notion  of  polar  factorization  which  allows  one  to  remove  the  curl  from  a 
vector  field  in  computing  the  optimal  transport  map.  These  ideas  can  naturally  be  used  in 
studying  optical  flow  as  is  illustrated  in  the  deforming  heart  image  given  below. 
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6.  Summary,  Conclusions,  and  Recommendations 


The  phase  I  effort  documented  herein  produced  an  innovative  and  successful  method  for 
direct  adaptive  guidance  for  intercept  of  a  maneuvering  target  without  the  use  of  a  model  of 
target  acceleration.  Furthermore,  the  formulation  allows  for  neural  network  augmentation  of  a 
variety  of  nominal  guidance  strategies,  such  as  those  that  arise  from  our  study  of  biological 
systems.  Feasibility  of  the  new  adaptive  guidance  method  was  clearly  demonstrated  in 
simulation  of  a  2-D  target  engagement. 

The  phase  I  effort  also  produced  a  very  successful  experimental  method  for  capturing  the 
trajectories  and  orientations  of  flies  in  pursuit  of  a  target,  and  clearly  demonstrated  the  feasibility 
and  potential  of  collecting  the  3-D  and  orientation  data  necessary  to  fully  characterize  the 
tracking,  guidance,  and  control  strategy  employed  by  the  flesh  fly. 

The  phase  I  effort  also  produced  improvements  to,  and  demonstration  of,  practical  image 
processing  algorithms  that  can  be  used  to  realize  the  envisioned  system  for  controlled  active 
vision  in  hardware  and  software.  The  algorithms  were  exercised  using  simulated  imagery  of  a  2- 
D  missile  engagement  with  proportional  navigation  against  a  maneuvering  target,  and  was 
successful  in  autonomously  selecting  and  tracking  the  target.  Live  demonstrations  of  this 
tracking  algorithm  were  presented  to  the  sponsor  at  the  last  program  review  meeting.  The 
computation  of  optical  flow  was  also  improved,  and  applied  to  digital  image  sequences  of  the 
male  flies  in  pursuit.  The  tracking  algorithm  was  shown  to  be  very  effective  in  this  application 
despite  the  cluttered  environment  and  obstacles  that  temporarily  obscured  the  flies  from  view.  In 
particular,  the  tracking  algorithm’s  ability  to  recognize  and  recover  tracking  after  the  fly  had  been 
temporarily  obscured  from  view  was  demonstrated. 

It  is  recommendd  that  these  diverse  technical  elements  be  combined  in  follow-on  research 
to  produce,  integrate,  test  and  demonstrate  practical  biologically-inspired  methods  for 
autonomous  guidance  and  control  that  engage  targets  in  densely-cluttered  environments  using 
vision  as  the  primary  sensor  modality. 
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